Method for controlling a terminal display and a terminal 



FIELD 

[0001] The invention relates to a method for controlling a display in 
a terminal, particularly in portable terminals, where the small size of the 
terminal restricts the size of the display and the user interface, and to a 
terminal. 

BACKGROUND 

[0002] At present the control of a display in telecommunication 
equipment, for example when playing different games, is generally based on 
using a mouse or different joysticks. When portable equipment such as 
handheld computers or phones are concerned, such solutions seem clumsy 
and impractical; a mouse may, for example, be of the same size as the entire 
apparatus. Likewise, if the motoric co-ordination or manual power of a terminal 
user have weakened due to injury or some other circumstance, such as age, 
the use of a mouse or joystick can be difficult. Therefore, a need is created to 
solve the display control of portable telecommunication equipment in particular 
in some other ways. 

[0003] As component technology advances, new components have 
appeared on the market, such as acceleration sensors and proximity sensors. 
An acceleration sensor is generally an electromagnetic component generating 
to its output an electric signal corresponding to the acceleration. An 
acceleration sensor may presently be implemented in various ways, for 
example using a piezoelectric or piezoresistive crystal. The acceleration 
sensors may also be based on the motion of the mass. In such a case the 
motion of this mass is measured, for example capacitively, in order to 
determine the acceleration. 

[0004] A change in the charge distribution of the piezoelectric 
crystal is comparable with the strength put upon the crystal. A piezoelectric 
crystal converts the mechanical labour into electricity and vice versa. In the 
piezoresistive crystal, in turn, the electric charge of the crystal is comparable 
with the power put upon the crystal. 

[0005] The proximity sensors can be implemented using optical 
components. The optical proximity sensors are based on sending and 
receiving light of different wavelenghts. The focused light sent from a 
transmitter is reflected back from the target, the receiver receives the reflected 
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signal and converts the signal to distance information, for example. The 
proximity sensors can also be implemented using other techniques, for 
example, pyro sensors sensitive for thermal radiation. 

BRIEF DESCRIPTION 

5 [0006] It is an object of the invention to provide an improved 

method for controlling a terminal display. 

[0007] This is achieved with a method for controlling a terminal 
display. The method comprises the steps of providing the terminal display with 
P at least one virtual display, a display portion or an object, and moving at least 

O 10 the virtual display, the display portion or the object on the display using the 
motion of the terminal or the motion and/or location of an object proportioned 
^ to the terminal. 

[0008] The invention also relates to a terminal implementing the 
method, the terminal comprising a display and a user interface. The terminal 
15 comprises means for providing the terminal display with at least one virtual 
Uj display, a display portion or an object, and the terminal comprises means for 

moving at least the virtual display, the display portion or the object on the 
display using the motion of the terminal or the motion and/or location of an 
object proportioned to the terminal. 
20 [0009] The preferred embodiments of the invention are disclosed in 

the dependent claims. 

[0010] The invention is based on the idea that the terminal display 
is provided with a virtual display, a display portion or an object, which are 
moved by means of the motion of the terminal or the motion and/or location of 
25 an object proportioned to the terminal. The object that is moved in relation to 
the terminal, thus controlling the terminal display, is preferably a hand, but 
may also be another object, such as an instrument intended for handicapped 
or elderly people. 

[0011] The method and arrangement of the invention provide 
30 several advantages. The method of the invention allows to control the display 
of portable terminals without external user interfaces connected to the 
terminal, such as a mouse or a joystick. The virtual display created on the 
physical display allows to play games using portable equipment. Likewise, if 
the size of the actual display exceeds the one that can be fitted onto a physical 
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display, then the physical display can be used to show only a^part of the actual 
display and to browse the display rapidly and without accessories. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] In the following the invention will be described in greater 
detail by means of the preferred embodiments with reference to the 
accompanying drawings, in which 

[0013] Figure 1 shows an example of a telecommunication system, 

[0014] Figure 2 illustrates an example of a terminal, 

[0015] Figure 3 is a flowchart showing the method steps for 
controlling a terminal display, 

[0016] Figure 4 illustrates an example of how acceleration sensors 
are used in a terminal, 

[0017] Figure 5 illustrates an example of how proximity sensors are 
used in a terminal, 

[0018] Figure 6 shows an example of a virtual display in a shooting 

game, and 

[0019] Figures 7a to c show an example of how proximity sensors 
are positioned and a model of how a bow is drawn and released in an archery 
game. 

DESCRIPTION OF EMBODIMENTS 

[0020] Figure 1 illustrates in a simple manner a digital 
telecommunication system to which the solution of the invention can be 
applied. A part of a cellular radio system is concerned that comprises a base 
station 104, communicating over a radio connection 108 and 110 with 
subscriber terminals 100 and 102, which may be fixedly positioned, vehicle 
mounted or portable terminals. The base station includes transceivers. The 
base station transceivers communicate with an antenna unit, which is used to 
implement the radio connection to the subscriber terminal. The base station 
also communicates with a base station controller 106 that forwards the 
terminal connections to the rest of the network. The base station controller 
centrally controls several base stations communicating therewith. A control 
unit in the base station controller carries out call control, mobility management, 
statistical data assembly and signalling. 

[0021] The cellular radio system may also communicate with a 
public telephone network. 



[0022] Figure 2 illustrates an example of a terminal, where 
acceleration sensors, proximity sensors, OTM sensors or corresponding 
components are utilized for controlling the display. The terminal may for 
instance be a portable phone or a microcomputer, however, without being 
restricted thereto. 

[0023] The terminal described comprises an antenna 200, which is 
used to send and receive signals through a duplex filter. The terminal also 
comprises a transmitter 202 that amplifies and sends the modulated signal to 
the antenna, a modulator 204 that modulates the carrier wave with a data 
signal including the desired information in accordance with a selected 
modulation method, a receiver 206 that amplifies the signal arriving from the 
antenna and downconverts it to the selected intermediate frequency or directly 
to the baseband, a demodulator 208 that demodulates the received signal, so 
that a data signal can be distinguished from the carrier wave, and a controller 
part 216 comprising, for example, control and calculation means for controlling 
the operation of the different terminal parts and means for processing the 
user's speech or the data generated by the user, such as a DSP processor 
(Digital Signal Processing) comprising, for instance, channel equalizer 
operations that compensate for the interferences caused by the radio channel 
signal utilizing the data obtained from the channel using a known training 
sequence, A/D converters that convert the analogue signal into digital mode by 
sampling and quantizing the baseband signal, and D/A converters that convert 
the digital signal into analogue mode using a reversed method, filters that filter 
the frequencies outside the desired frequency band in the receiver or that 
restrict the bandwidth of the output in the transmitter in band-limited systems, 
and encoding and decoding means performing both channel and speech 
coding. 

[0024] In channel coding, the systematic bit redundancy added to 
the signal, typically parity bits, is used in the decoder for detecting and 
correcting errors. In speech coding, in source coding in general, the 
unsystematic redundancy in the source symbols is typically removed in order 
to reduce the required bit rate. In addition, in spread spectrum systems, such 
as the WCDMA, the signal spectrum is spread in the transmitter using a 
pseudo-random spreading code to a broad band, and the signal spectrum is 
despread in the receiver, thus attempting to increase the channel capacity. 
Coding may be employed for enciphering the output or the information therein. 



Furthermore, the controller part typically comprises in the GSM system 
apparatuses burst generation means, which add the tail bits of the burst and 
the training sequence to the data arriving from the channel codec. The 
controller part also comprises means for arranging the signal to be sent as well 
as the signalling information to be in line with the air interface standard of the 
cellular radio system in use. The controller part 216 also comprises means for 
processing sensor signals and for controlling the display. The above terminal 
is a terminal in a digital cellular radio system, but the invention may 
correspondingly be applied to analogue systems. 

[0025] The user interface of the terminal comprises a loudspeaker 
or a headphone 210, a microphone 212, a display 218 and possibly a 
keyboard communicating with the controller part. The keyboard may also be a 
touch control switch. The display may be a black or white one or a colour 
display, and the image on the display may either be motionless or moving. The 
terminal also comprises one or more acceleration or proximity sensors or 
corresponding components 220 for controlling the display. The sensors allow, 
for example, to detect the location or motion of the hand or another object in 
relation to the display. The terminal may also comprise a camera for taking 
images in order to identify motion. The terminal also comprises various 
memory elements, which are presented as a single functional^ block 214. The 
memory element includes, for example stored data, such as messages arriving 
to the user from the network part of the system. A part of the memory element 
can also be employed as a memory buffer. The memory element also includes 
a program controlling the operation of the terminal and comprising, for 
example, sub-programs that typically handle different tasks associated with the 
control of the display. The operations according to the invention for processing 
the sensor signals and for controlling the display can typically be implemented 
by software by including software comprising the required commands to be 
used by the control unit of the terminal. 

[0026] Figure 3 is a flow chart showing the method steps for 
controlling the display of the terminal. The terminal display can be controlled 
when the terminal is, for example, in the user's hand, on the user's lap or on 
the table. The method starts from block 300. In block 302 at least one virtual 
display, a display portion or an object, which can be moved, is created for the 
physical display of the terminal. In block 304, at least the single virtual display, 
the display portion or the object on the display is moved using the motion of 



the terminal or the motion and/or location of an object proportioned to the 
terminal. The motion of the terminal or the motion or location of an object, 
such as a hand or an instrument for handicapped or elderly people in relation 
to the terminal, is observed using acceleration sensors, proximity sensors or 
other kinds of generally optical sensors, such as OTM sensors or using a 
camera to take at least two consecutive images, and identifying the motion by 
comparing said images. 

[0027] An individual acceleration sensor element reacts to the force 
from one direction only. Acceleration can be measured in several directions 
using the acceleration sensors, which are composed of two or three 
acceleration sensor elements. Furthermore, if the terminal comprises more 
than one sensors measuring in three dimensions, then the changes of the 
apparatus position, such as turning the apparatus into a particular direction, 
can be observed in addition to the motion of the terminal. 

[0028] The proximity sensors are, in turn, typically optical 
components based on sending and receiving an optical (IR, visible light, UV) 
signal. The proximity sensors may also be based on heat radiation. The 
proximity sensors may be based on a transmitter-receiver pair, in which case 
the response of the receiver changes as a function of the distance of the 
object. 

[0029] An OTM sensor is a component type, which is able to 
measure the motion of a surface in relation to the OTM sensor, developed by 
the component manufacturer OTM Technologies LTD. An OTM sensor is 
capable of measuring the proximity of a surface or the transverse motion 
across the aperture of the sensor. In an embodiment of the method where the 
OTM sensor is used, an object can be moved on the terminal display, such as 
a cursor or a finder in game applications. In such a case, two sensors are 
used for locating or moving the object to be moved in the vertical or horizontal 
direction. A third sensor allows to determine the quality of the surface beneath 
the sensor. 

[0030] The view of the terminal display can also be scrolled, 
meaning that only a part of the actual view is shown at a time on the display, 
and the actual view may be significantly larger than the one shown on the 
display. Thus, when playing a game for instance, the background of the game 
or the appearance of the butt, such as size, model and/or colour can be 
changed. The view can be scrolled by informing the terminal about at least 
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one location point as it were the origin of the coordinate system. Thereafter by 
moving the terminal or an object in relation to the terminal, the view on the 
display is scrolled correspondingly. Then, by moving the terminal upwards, the 
view of the display also moves upwards, and the terminal display shows an 
upper portion of the previous actual view. The location point can be indicated 
for example by lifting up the terminal and by determining this position as the 
uppermost location by selecting the position (up) from the menu or by using 
speech identification and saying "up" or using another appropriate way. A 
coordinate system or a block division resembling the one used in maps may 
also be employed when indicating the location point. 

[0031] Arrow 306 illustrates the repeatability of the method starting 
from the creation of the virtual display. Arrow 308 illustrates the repeatability of 
the method using the previously created virtual display, whereby the virtual 
display, a portion thereof or another object, such as a finder or a cursor, is 
moved as frequently as necessary. 

[0032] The method ends at block 310. 

[0033] Figure 4 illustrates the use of acceleration sensors in a 
terminal using shooting and archery game examples. The acceleration sensor 
in block 400 observes the motion of the terminal or of an object in relation to 
the terminal in the horizontal direction. The acceleration sensor in block 402 
observes the motion of the terminal or another object in relation to the terminal 
in the vertical direction, and the acceleration sensor in block 404 observes the 
motion of the terminal or another object in relation to the terminal in the depth 
direction. Thus, the motion can be observed in a three-dimensional space. The 
signals of sensors 400, 402, 404 are supplied to the A/D converter 406, where 
the signals are sampled and quantized in order to convert them into digital 
mode for signal processing. 

[0034] A filter 408, which typically is a low-pass filter, is used to filter 
the signals caused by too rapidly changing motions, such as the trembling of 
the hand, from the sensor signals. A targeting point on the display can thus be 
kept more stable. The remaining signal comprises the positioning data of the 
apparatus. Figure 6 shows an example of a virtual butt 602 and a targeting 
point 604 created onto the physical display 218. The targeting point can be 
indicated, for example, with a cross, point or arrow. The butt can also be 
moved using a computer. 
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[0035] In block 414 the orientation of the terminal is calculated in 
relation to the calibrated direction on the basis of the sensor signals typically 
by means of a microprocessor. This enables the continuation of the game 
when the terminal is in different positions. The orientation may be selected by 
5 the player himself/herself or it can be automatically determined. In block 416 
the tilt, acceleration, angle and rotation of the virtual display, the display 
portion or the object on the display or the motion thereof are correspondingly 
calculated in relation to the calibrated positions. The tilt is calculated in the 
forward, backward and side directions. The data determined in block 416 are 
10 used for controlling the finder. 

[0036] Block 410 estimates the stability of the terminal, for example, 
by calculating the variance of the signal in different directions using the sensor 
signals and the motion calculated in block 414. The stability data can be used, 
for instance, for controlling the finder. It is not necessary to calculate the 
jp 1 5 stability. 

* s [0037] The degree of difficulty is determined in block 412. The 

pj game becomes more difficult, if the too rapidly changing motions, such as the 

trembling of the hands, are less filtered, whereby the aiming point, for 
example, trembles with the hand. In such a case, it obviously becomes more 
20 difficult to hit the target. It also becomes more difficult to hit the target, if the 
size of the target is reduced. If the degree of difficulty of the game is changed, 
the change can be carried out by the player himself/herself or the change can 
be programmed to take place, for example at certain intervals, or when the 
player has received such a number of points in the game that exceeds the 
25 predetermined threshold value. There may be more than one difficult degree 
thresholds. 

[0038] Firing is carried out in block 418 using a key in the terminal 
or a touch control switch. The firing can also be carried out using a sound 
signal. The player is informed about the firing with a sound signal by means of 
30 a loudspeaker, with a light signal seen on the display or with a vibrator. 

[0039] Signal processing of the sensor signals as well as the control 
of the operations, such as determining the game's degree of difficulty, are 
carried out in block 216, and commands are conveyed to the display, i.e. the 
display is controlled. 

35 [0040] In the following, Figures 5 and 7a to c describe an example 

on how the proximity sensors are used in the terminal when playing an archery 



jjssss 



game. Figure 5 is a flowchart showing an example of a simplified terminal. 
Figure 7a is a diagram showing an example of how the proximity sensors can 
be placed into the terminal. Proximity sensors are generally optical 
components. Proximity sensors 500 and 502 are placed in a terminal 700 so 
that they are in a most appropriate position for playing the archery game. In 
Figure 7a the sensors are placed at the lower edge of the display 218. In 
general, the terminal also comprises keys or touch control switches 704. The 
stretching and release force of the bow are determined by means of the 
proximity sensors. The proximity sensors may be based on the transmitter 
receiver pairs, for example, whereby the response of the receiver changes as 
the function of the object's distance. The proximity sensors are typically optical 
components, and the signal to be transmitted is thereby light, for example, 
within the area of the carrier wave of the visible light. For instance, the optical 
sensor sends light and measures the amount of the back-reflected light. If an 
object obstructs the view, then only a part of the light intensity is reflected back 
from the object. The proximity sensor may also be based on utilizing heat 
radiation. 

[0041] The proximity sensor signals are processed in block 504. 
According to the example shown in Figure 5, this block is provided with, for 
example, an A/D converter and a filter that is typically a low-pass filter. The 
filter is used for filtering signals caused by the too rapidly changing motions, 
such as trembling of the hands, from the sensor signals. In this way the 
trembling of the butt on the display can be adjusted. Block 504 also comprises 
the compensation of the surrounding light. The ambient light compensation 
must be carried out in order for the optical proximity sensors to operate 
correctly irrespective of the lighting conditions. 

[0042] In block 506 the stretching force is typically estimated using 
a microprocessor by determining a first object or in this case the distance of a 
finger from a first proximity sensor, and the distance of a second object, or in 
this case another finger, from a second proximity sensor. Figure 7b shows an 
example of how to stretch the bow; objects 710, 712, in this case preferably 
the fingers, approach one another. The bow is stretched to the maximum, 
when the fingers touch. The terminal user can be informed about the stretch of 
the bow, for example, graphically on the display or using a sound signal. 

[0043] The release force is estimated in block 508 in accordance 
with Figure 7c; the further the objects 710, 712 are from one another in Figure 



7c, the greater the release force becomes. The recognition of the stretch and 
release of the bow is based on the change of the proximity senor signals. The 
release occurs by identifying the two objects, preferably the fingers, which 
move apart from one another. The release can be carried out also using a 
sound signal or touching a key. The player is informed about the release with 
a sound signal using a loudspeaker, with a light signal seen on the display or 
with a vibrator. 

[0044] Blocks 506 and 508 may also comprise pattern recognition 
in accordance with a known method. In pattern recognition a pattern of an 
object, such as outlines, is typically formed using the signal generated for the 
proximity sensors of an object, preferably the fingers, in the archery game 
example. The formed patterns enable to recognize the functions associated 
with the game, such as stretching and releasing the bow. 

[0045] The control block 216 is in charge of processing the sensor 
signals, determining the forces and transmitting commands onto the display. 

[0046] The invention is preferably implemented by software, the 
terminal thus comprising one or more microprocessors, in which the 
operations of the method presented as the operating software are 
implemented. The invention can also be implemented, for example, with 
apparatus solutions offering the required functionality, such as ASIC 
(Application Specific Integrated Circuit) or by utilizing separate logic 
components. 

[0047] Even though the invention has above been explained with 
reference to the example in the accompanying drawings, it is obvious that the 
invention is not restricted thereto but can be modified in various ways within 
the scope of the inventive idea disclosed in the attached claims. 



